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Abstract 


We  present  results  from  an  ongoing  investigation  using  a  work-centred  framework  to  design 
computer-based  tools  to  support  the  cognitive  and  collaborative  work  of  Command  Teams  on  a 
FIALIFAX  Class  frigate.  Based  on  emerging  concepts  in  Cognitive  Systems  Engineering,  the 
design  approach  hinges  on  analyzing  the  operators’  work  demands  and  finding  ways  to  use 
technology  to  improve  the  performance  of  the  joint  cognitive  system  of  operators  and  their  aids. 
Specifically,  we  describe  our  use  of  two  work-modeling  tools  within  the  framework  that 
contribute  to  developing  a  traceable  design  thread,  directly  linking  knowledge  elicitation  and 
work  analysis  outputs  to  specific  design  hypotheses  for  supporting  the  work  demands  of 
Command  and  Control  operators.  The  two  tools  are  drawn  from  Rasmussen  and  Vicente’s 
Cognitive  Work  Analysis  framework.  The  first  is  an  Abstraction-Decomposition  Space.  It  uses 
an  Abstraction  Hierarchy  to  describe  how  the  functional  purposes  of  the  work  domain  are 
achieved.  There  is  also  a  Part-Whole  Hierarchy  that  decomposes  the  work  domain  into 
components  that  contribute  to  achieving  those  purposes.  The  second  tool  is  a  set  of  decision 
ladders  representing  the  information  processing  and  knowledge  states  of  operators  in  conducting 
control  tasks.  We  also  briefly  illustrate  one  interface  concept  that  was  derived  with  the 
approach. 

1 .  Introduction 

Defence  R&D  Canada  -  Atlantic  is  investigating  technologies  such  as  data  fusion  and  advanced 
operator-machine  interfaces  to  support  naval  operators  in  the  Operations  Room  of  a  HALIFAX 
Class  frigate  in  their  Command  and  Control  (C2)  work,  focusing  specifically  on  their  work  areas 
of  maritime  tactical  picture  compilation  (MTPC)  [1]  and  tactical  planning  and  response 
management  (TPRM)  [2],  The  tactical  picture,  compiled  from  Above  Water  Warfare, 
Underwater  Warfare,  Tactical  Data  Link  and  Wide  Area  Picture  data  inputs,  is  the  situation 
picture  underlying  all  aspects  of  Command  and  Control  decision  making  over  an  area  of  interest 
of  a  maritime  commander.  Tactical  planning  and  response  management  is  concerned  with  the 
development,  integration  and  management  of  tactical  plans  within  and  across  the  air,  surface  and 
subsurface  warfare  areas. 


A  key  aspect  of  this  investigation  is  determining  the  requirements  for  effective  computer-based 
tools  for  MTPC  and  TPRM.  The  research  literature  on  existing  complex  sociotechnical  work 
environments  that  share  many  of  the  characteristics  of  C2  provides  strong  evidence  for  the  merit 
of  designing  cognitive  and  collaborative  support  systems  for  such  environments  based  on  an  in- 
depth  understanding  of  the  work  system  and  the  specific  work  demands  that  operators  have  to 
deal  with  (e.g.,  [3]).  In  such  a  design  approach,  some  form  of  work  analysis  to  model  the 
operators’  work  demands  posed  by  the  work  environment  inevitably  emerges  as  a  critical 
consideration.  We  have  been  investigating  such  an  approach  for  the  HALIFAX  Class  frigate.  It 
is  based  on  emerging  concepts  in  the  field  of  Cognitive  Systems  Engineering  [3]. 

In  this  paper,  we  review  our  design  framework  and  describe  how  two  work-modeling  tools  have 
been  used  within  the  framework  to  identify  work  demands  of  C2  operators  on  a  HALIFAX  Class 
frigate  and  uncover  potential  design  solutions  to  support  these  demands.  A  significant  and  novel 
contribution  of  the  approach  is  that  it  permits  developing  a  traceable  design  thread  that  directly 
links  knowledge  elicitation  and  work  analysis  outputs  to  specific  design  hypotheses  for 
supporting  operator  work  demands.  Finally,  we  briefly  illustrate  one  tentative  interface  concept 
for  supporting  tactical  picture  compilation  work  on  the  frigate  that  was  derived  with  the 
approach. 

2.  Exploratory  design  framework 


Evaluated 

Design 

Seed 


Figure  1.  Methodological  framework  for  eliciting  design  seeds 

The  primary  purpose  of  our  exploratory  design  framework  is  to  permit  developing  and  testing, 
from  a  work-centred  perspective,  design  hypotheses  for  supporting  operators  with  advanced 
computer-based  capabilities  in  their  individual  or  collaborative  cognitive  work.  We  review  its 
principal  elements.  Additional  details  can  be  found  in  [1].  The  framework  encompasses  various 


activities  aimed  at  developing  an  increasing  understanding  of  the  work’s  demands,  and  using  this 
to  develop  and  test  specific  hypotheses  about  ways  to  support  operators  with  these  demands. 
Figure  1  illustrates  the  framework,  showing,  for  concreteness,  one  specific  activity  trajectory 
within  it,  and  the  activity  nodes  on  that  trajectory  and  their  potential  linkages  in  tenns  of  inputs 
and  outputs.  In  practice,  however,  design  involves  continuously  shifting  focus  in  a  nonlinear 
manner  among  the  activities  in  Fig.  1,  between  the  problem  space  (the  work  environment)  and 
the  solution  space  (the  design  space),  according  to  a  hypothesize-and-test  design  paradigm. 

An  essential  part  of  the  framework  is  a  work  analysis  that  explicitly  models  work  demands  or 
constraints  in  the  work  environment  as  a  basis  for  design.  This  explains  why  we  refer  to  the 
approach  as  work-centred.  Analysis  is  therefore  a  work-modeling  process,  providing  a  critical 
link  between  knowledge  acquisition  and  design  activities.  This  analysis  is  conducted  along  the 
lines  of  Rasmussen  and  Vicente’s  Cognitive  Work  Analysis  (CWA)  framework  [3],  CWA  is  a 
systems-oriented  approach  to  analyzing  a  work  environment  aimed  at  capturing  the  behaviour¬ 
shaping  constraints  for  that  environment.  Such  constraints  delimit  an  envelope  within  which  all 
productive  work  occurs.  This  underlies  the  formative  focus  that  CWA  brings  to  design,  by 
providing  a  modeling  capability  to  deal  with  demands  across  a  broad  spectrum  of  situations, 
from  familiar  ones  that  operators  encounter  routinely,  to  unfamiliar,  but  anticipated  ones  (i.e., 
anticipated  by  system  designers,  policy,  doctrine,  tactics,  and  procedures),  to  ones  that  are 
unfamiliar  and  unanticipated. 

Hypotheses  for  supporting  work  are  rendered  as  design  seeds,  each  seed  effectively  instantiating 
some  specific  support  hypothesis  that  needs  to  be  tested1.  Figuratively,  work  analysis  serves  to 
‘seed’  promising  design  concepts  that  may  turn  into  design  ‘nuggets’,  which  when  integrated 
could  be  used  in  defining  a  complete  capability  for  supporting  work  demands.  Testing  the 
validity  of  a  support  hypothesis  for  a  seed  might  range  from  obtaining  initial  subjective  Subject 
Matter  Expert  (SME)  feedback  to  the  seed  to  conducting  objective  performance  tests  with  it.  In 
this  manner,  therefore,  complex  aspects  of  the  work’s  demands  are  decomposed  into  manageable 
portions  for  design  purposes.  In  addition,  increasingly  realistic  prototypes  of  design  seeds  can 
be  iteratively  developed,  refined,  integrated,  and  tested,  leading  eventually  to  a  coherent  support 
capability. 

We  have  used  a  variety  of  approaches  in  our  work  to  date  to  do  the  knowledge  elicitation  with 
SMEs  to  produce  domain  knowledge  needed  for  the  CWA  modeling  shown  in  Fig.  1.  In  one 
approach,  described  in  [1],  a  detailed  paper-based  description  of  a  tactical  scenario  was  first 
prepared.  Teams  of  operators  were  then  walked  through  a  series  of  static  screen  configurations 
presenting  the  state  of  the  tactical  situation  confronting  the  frigate  at  various  times  while  they 
described  their  own  and  their  team’s  activities,  in  terms  of  their  goals,  infonnation  needs, 
information  sources,  information  transfers,  processing  activities,  strategies,  and  the  collaboration 
that  might  be  involved.  Concerns  with  this  approach  stemmed  largely  from  its  inability  to 
capture  the  rich  detail  and  dynamic,  contextual  nature  of  C2  work  on  the  frigate. 


1  Referring  to  these  instantiations  as  design  seeds  is  meant  to  capture  their  essential  role  as  one  of  seeding  or 
jumpstarting  an  exploratory  design  process.  The  terminology  is  borrowed  from  Patterson  et  al.  [4].  A  seed 
represents  some  specific  and  relatively  independent  support  concept  for  some  specific  aspect  of  the  work.  In  our 
work,  we  have  focused  on  seeds  of  a  computer-based  nature.  A  particular  seed  could  be  realized  in  a  number  of 
forms,  from  a  very  crude,  static  mock-up  to  some  sort  of  working  prototype. 


In  subsequent  work,  we  adopted  a  detailed  scenario-based  approach  [5].  Significant  effort  was 
expended  to  work  with  an  SME  to  first  build  a  complex,  tactical  scenario  comprised  of  a  variety 
of  challenging  work  situations  set  in  a  realistic  mission  context.  This  was  then  programmed  and 
run  in  a  Canadian  Navy  trainer  with  a  portion  of  the  Operations  Room  operators  working  as  they 
would  normally  in  response  to  the  unfolding  events  in  the  scenario.  Data  was  collected  during 
scenario  runs,  at  pauses  during  intervals  when  the  scenario  was  frozen,  and  then  at  the  end  of  a 
run  in  a  debrief  session.  While  this  approach  remedied  some  of  the  deficiencies  of  the  previous 
paper-based  method,  it  also  exposed  some  critical  data  collection  deficiencies  in  the  trainer. 

Recent  knowledge  elicitation  has  focused  on  adapting  Klein’s  Critical  Decision  Method  (CDM) 
[6].  This  change  stemmed  from  a  concern  that  by  presenting  SMEs  with  a  scenario,  it  restricted 
the  breadth  of  information  that  could  be  collected.  This  would  mean  that  any  support  concept 
eventually  developed  would  address  a  ‘layperson’s’  understanding  of  what  the  naval  operator 
does,  rather  than  the  SME’s.  Because  the  layperson  cannot  have  the  SME’s  breadth  of  domain 
experience  and  understanding,  issues  identified  and  their  associated  support  concepts  might  not 
address  the  more  critical  operator  work  demands.  To  overcome  this  in  scenario-based  data 
collection  approaches,  an  ‘open’  approach  to  data  collection  (CDM)  was  therefore  adopted. 

CDM  has  pennitted  SMEs  to  describe  what  they  considered  to  be  the  most  difficult  situations 
they  encountered.  Once  the  problem  space  was  bounded  in  this  way,  analysts  could 
systematically  investigate  each  facet  of  the  problem  space.  This  approach  provided  the  data  to 
construct  a  detailed  account  of  the  actions  the  operators  would  undertake  in  the  scenario  they 
described,  including  the  triggers,  information  requirements  and  outputs.  Using  this  approach,  it 
is  felt  that  the  resulting  support  concepts  will  be  more  relevant  and  acceptable  to  operators. 

It  is  unlikely  that  any  one  knowledge  elicitation  approach  will  provide  all  the  data  needed  to 
conduct  a  complete  CWA.  Multiple  techniques  may  be  needed,  tailored  to  the  specific  types  of 
demands  being  modeled  or  the  specific  design  questions  being  considered,  or  simply  to  produce 
a  converging  picture  of  those  demands.  There  will  also  be  a  variety  of  pragmatic  constraints 
such  as  the  availability  of  SMEs,  the  time  that  can  reasonably  be  devoted  to  data  collection,  the 
cost  and  overhead  involved,  and  so  on,  that  enter  into  the  choice  of  method(s).  Further  research 
is  needed  to  improve  our  understanding  of  the  large  variety  of  considerations  involved  and  the 
various  approaches  that  could  be  productively  employed  in  conducting  a  CWA. 

3.  Work  modeling 

In  this  section  and  the  next,  we  discuss  how  we  have  been  using  the  previously  described  design 
framework  to  elicit  design  seeds  according  to  the  design  activities:  develop  work  models  -> 
develop  design  seeds  and  their  associated  support  hypotheses.  We  concentrate  on  the  specific 
process  employed  in  recent  work  to  elicit  design  seeds  for  TPRM,  where  a  variant  of  the  CDM 
was  used  with  three  teams  of  operators  over  a  period  of  1 .5  days  in  the  Knowledge  Acquisition 
activity  in  Fig.  1. 

The  current  section  focuses  specifically  on  the  development  of  the  work  models  from  the  CDM 
data.  Two  types  of  work  models  have  been  developed  by  a  team  of  four  analysts:  a  Work 
Domain  Analysis  (WDA)  and  a  Control  Task  Analysis  (CTA).  Both  are  steps  in  a  CWA  [3].  A 
WDA  identifies  the  functional  and  part-whole  structure  of  the  work  domain.  A  CTA  identifies 
what  needs  to  be  done  in  effecting  control  tasks  in  the  work  domain. 


3.1  WDA  modeling 

A  WDA  models  a  work  domain  in  the  form  of  an  Abstraction-Decomposition  Space  (ADS), 
displayed  as  a  matrix.  Along  the  vertical  axis  of  the  matrix  are  the  various  abstraction  levels  for 
the  work  domain  (its  Abstraction  Hierarchy).  Along  the  horizontal  axis  is  a  Part- Whole 
Decomposition  of  the  domain  corresponding  to  its  different  levels  of  resolution. 

Five  levels  of  abstraction  were  used.  These  are  shown  in  Table  1,  along  with  the  generic 
questions  each  level  provides  answers  to  for  the  domain  [7].  Based  on  SME  statements  collected 
in  the  CDM,  analysts  developed  a  list  of  decomposition  levels  that  were  referred  to  and  which 
were  therefore  involved  in  some  way  in  the  work  of  the  Operations  Room  team.  Seven  levels  of 
Part- Whole  Decomposition  were  thereby  established,  and  descriptions  of  what  each  level 
encompassed  were  created.  These  levels  and  their  descriptions  are  shown  in  Table  2. 


Abstraction 
Hierarchy  Level 

Generic  Questions 

Functional  Purpose 

What  was  the  work  domain  designed  to  do? 

Abstract  Function 

What  are  its  underlying  laws  or  principles? 

Generalized  Function 

What  are  the  processes  that  are  involved? 

Physical  Function 

What  entities  are  involved  and  what  are  their  capabilities? 

Physical  Form 

What  is  the  physical  appearance  and  location  of  an  entity? 

Table  1.  Levels  in  the  Abstraction  Hierarchy 


World  Operational  Local  System  Sub-system  Component  Sub- 

Environment  Environment  Component 


Geopolitical, 

Physical  (including 

Logical 

Self-contained 

Logical 

groupings 

Entities  within 

Component 

weather, 

air,  surface  and 

groupings  within 

units  within  the 

within 

self- 

logical 

elements  of 

geophysical, 

subsurface  contacts 

operational 

operational 

contained  units 

groupings  of 

entities  (e.g.,  a 

etc. 

-  both  hostile  and 

environment 

environment 

(e.g., 

personnel, 

self-contained 

weapon,  an 

friendly  (Task 

Group))  and  non¬ 
physical  (air/ship 

ianes,  weather) 

elements 

(e.g.,  Task 

Group,  Air 

Contacts, 
Environment) 

(e.q.,  Own- 

Ship) 

vehicles, 

systems) 

ship 

unit  (e.g., 

information 

system, 

communication 

system,  bridge 

personnel, 

helicopter, 

weapons 

systems) 

element  of 

database, 
information, 
rudder) 

Table  2.  Part-whole  decomposition 


Once  the  ADS  matrix  was  created,  each  SME  statement  was  mapped  independently  by  each 
analyst  to  a  specific  cell  in  the  ADS.  A  set  of  exemplars  was  also  produced  for  each  cell  of  the 
ADS.  An  exemplar  is  a  particular  example  of  a  type  or  category  of  a  member  of  the  cell.  For 
example,  “Meet  Naval  Values”  was  an  exemplar  for  the  Functional  Purpose  of  the  System 
(comprising  OwnShip).  A  starting  point  for  the  exemplars  was  earlier  work  to  develop  an 
Abstraction  Hierarchy  for  the  frigate  [8].  Exemplars  provided  a  consistency  check  for  analysts 
to  individually  determine  whether  their  mapping  of  SME  statements  to  ADS  cells  was 
appropriate.  Analysts  also  met  at  the  end  to  discuss  their  analysis  and  agree  upon  a  final 


mapping.  If  any  clarification  was  needed  for  a  particular  statement,  its  wording  was  changed  to 
reflect  the  analysts’  interpretations.  During  these  discussions,  additional  mapping  rules  were 
also  discussed  and  clarified  as  needed.  This  mapping  onto  the  ADS  explicitly  exposed  how 
SMEs  traversed  the  different  levels  of  abstraction  and  decomposition  of  the  work  domain  over 
the  course  of  the  scenario  developed  during  the  CDM  interviews. 


Figure  2.  Decision  Ladder  template,  coding  scheme,  and  stages  of  human  information  processing 


For  a  concrete  example  of  the  mapping  process,  consider  the  SME  statement:  “Determine  trade¬ 
off  to  defend  them  and  put  self  at  undue  risk  (i.e.,  sacrifice  OwnShip  to  protect  a  High  Value 
Unit  (HVU))”.  In  this  statement,  ‘them’  refers  to  the  HVU  under  escort.  This  statement  impacts 
both  the  frigate  and  the  HVU.  The  decomposition  level  must  therefore  be  above  the  System 
level,  i.e.,  Local  Environment  or  Operational  Environment.  Since  the  HVU  was  part  of  the  Task 
Group,  this  statement  was  placed  at  the  Local  Environment  level.  Balancing  the  risks  involved 
in  tactical  plans  concerns  the  underlying  laws  or  principles  governing  operator  action.  There  is 


no  definite  process  governing  this  activity.  Therefore,  this  statement  was  placed  at  the  Abstract 
Function  level. 

3.2  CTA  modeling 

A  CTA  maps  out  data-processing  activities  and  states  of  knowledge  in  a  control  task,  from 
activation  to  execution,  in  the  form  of  a  ‘Decision  Ladder’  template.  The  ladder,  shown  in 
Figure  2,  has  been  ‘folded’  to  reinforce  the  notion  that  an  operator  can  take  efficient  processing 
‘shortcuts’  in  a  particular  task  by  ‘leaping’  or  ‘shunting’  between  non-sequential  points  in  the 
ladder,  thereby  bypassing  intervening  information-processing  steps.  Boxes  correspond  to  data- 
processing  activities  and  circles  to  states  of  knowledge. 


Decision  Ladder  Step 

Vicente  [3] 

Analysts’  Additional 
Interpretations 

Activation 

Detection  of  need  for  action 

Perception 

Alert 

What’s  going  on? 

Realisation 

Observe 

Information  and  data 

Display  of  contacts  and  other 
information 

Set  of  Observations 

What  lies  behind? 

A  body  of  information 

Identify 

Present  state  of  the  system 

Consider  the  information 

System  State 

What's  the  effect? 

What  does  this  mean? 

Interpret 

Consequences  for  current 
task,  safety,  efficiency,  etc. 

How  does  this  fit  into  the 
perceived  ‘idealised’  progress 
toward  the  goal? 

Ambiguity 

Which  goal  to  choose? 

It’s  unclear  how  it  fits 

Evaluate  Performance 

Criteria 

How  should  it  fit? 

Ultimate  Goal 

Which  is  then  the  goal  state? 

This  has  this  effect  on  the 
ultimate  goal 

Interpret 

Consequences  for  current 
task,  safety,  efficiency,  etc. 

What  steps  need  to  be  added  to 
get  progress  back  on  track? 

Goal  State 

Which  is  the  appropriate 
change  in  operating 
conditions? 

Know  what  needs  to  be 
achieved 

Define  Task 

Select  appropriate  change  of 
system  conditions 

Determine  what  need  to  be 
done 

Task 

How  to  do  it? 

Know  what  needs  to  be  done 

Formulate  Procedure 

Plan  sequence  of  actions 

Plan  how  to  do  it 

Procedure 

Know  how  to  do  it 

Execute 

Coordinate  manipulations 

Do  it 

Table  3:  Steps  in  Decision  Ladder,  with  additional  clarification 


To  simplify  the  analysis  of  SME  scenario  data,  a  coding  system  for  the  Decision  Ladder  was 
adopted  in  which  states  of  knowledge  corresponded  to  letters  and  data-processing  activities 
corresponded  to  numbers  (see  Fig.  2,  showing  the  codes  next  to  the  boxes  and  circles).  The 
‘Interpret’  activity  was  coded  twice  (4  and  6)  to  distinguish  whether  the  operator  is  interpreting 


the  consequences  of  the  ‘Ultimate  Goal’  or  interpreting  the  consequences  of  the  ‘System  State’. 
Coding  each  step  allowed  tracking  where  a  particular  decision-making  sequence  entered  the 
ladder,  how  it  moved  through  the  ladder,  and  where  it  left  the  ladder.  The  coding  structure  also 
permitted  easily  recognizing  leaps  and  shunts  in  a  particular  decision-making  sequence.  Shunts 
correspond  to  a  number  followed  by  a  non-sequential  letter  sequence,  and  leaps  to  two 
consecutive  letters. 

After  outlining  the  coding  structure,  further  definition  of  each  step  was  agreed  upon  by  all 
analysts.  This  ensured  that  analysts  had  a  consistent  interpretation  of  the  different  steps.  Table  3 
outlines  these  definitions.  Each  analyst  then  independently  analyzed  every  task  outlined  by  the 
SMEs.  Letters  and  numbers  were  assigned  in  order,  mapping  the  steps  involved  in  the  task  onto 
the  ladder.  As  in  the  WDA  modeling,  analysts  met  at  the  end  to  discuss  and  reconcile  the 
analysis  of  each  and  every  statement  and  to  agree  upon  a  final  mapping. 

To  concretely  illustrate  this  mapping  process,  consider  again  the  SME  statement  that  was 
previously  mapped  into  the  (Abstract  Function,  Local  Environment)  cell  of  the  ADS  matrix: 
“Determine  trade-off  to  defend  them  and  put  self  at  undue  risk”.  In  the  CTA,  this  was  mapped 
onto  the  sequence  4D5E6  in  the  Decision  Ladder.  Considering  the  task,  the  operator  (in  this  case 
the  ship’s  tactical  coordinator)  must  already  have  a  set  of  observations  and  know  the  system 
state;  he  is  now  using  this  information  to  “determine  trade-off...”.  The  operator  is  interpreting 
the  infonnation  he  has,  hence  ‘4’.  However,  because  he  is  detennining  trade-offs,  we  can 
assume  that  there  will  be  some  ambiguity,  an  assumption  supported  by  the  rest  of  the  statement 
“...to  defend  them  and  put  self  at  undue  risk...”.  This  takes  the  operator  to  the  ‘D’  state  of 
knowledge.  The  operator  must  then  evaluate  the  performance  criteria,  not  only  for  the  frigate 
(e.g.,  can  I  realistically  defend  myself  if  we  are  actually  attacked?  What  is  the  likelihood  of 
attack?),  but  also  for  the  mission  (e.g.,  do  I  really  have  to  defend  this  other  vessel?  What  are  my 
orders?).  This  leads  to  an  altered  ultimate  goal,  meaning  he  has  passed  through  ‘5’  on  his  way  to 
‘E’.  The  operator  is  left  considering  what  this  new,  altered  ultimate  goal  means  for  the  current 
task,  which  is  ‘6’  in  the  analysis. 

4.  The  process  of  developing  design  seeds  and  associated  support  hypotheses 

In  Section  3,  we  described  how  we  mapped  SME  statements  obtained  by  the  CDM  into  the 
Abstraction-Decomposition  and  Decision  Ladder  templates  of  a  WDA  and  a  CTA,  respectively. 
The  work  models  developed  in  this  manner  provide  the  basis  in  our  approach  for  considering 
what  is  uncovered  in  terms  of  design  seeds. 

A  design  seed  is  accompanied  by  a  hypothesis,  often  implicit,  regarding  the  nature  of  the  support 
it  affords.  For  instance,  a  seed  that  suggested  the  need  for  a  flashing  alert  to  cue  operators’ 
attention  would  carry  with  it  a  support  hypothesis  like:  the  operator  will  more  quickly  notice  a 
contact  in  the  presence  of  the  alert  than  without  it. 

We  found  it  useful  to  base  the  development  of  design  seeds  on  a  combined  set  of  considerations. 
Collectively,  these  considerations  can  be  viewed  as  stages  in  an  analysis  process  that  build  on 
each  other.  This  process  starts  with  an  initial  determination  of  a  generic,  coarse  set  of  design 
seed  themes  based  on  purely  cognitive  considerations,  and  converges  toward  a  set  of  specific 
design  seeds  and  their  associated  support  hypotheses  based  on  including  consideration  of  the 
results  of  the  WDA  and  CTA  modeling.  Four  stages  were  involved  in  the  process. 


1.  Identify  potential  difficulties  for  the  operator  on  the  basis  of  the  SME  statements 
collected. 

2.  Map  the  cognitive  basis  of  those  difficulties. 

3.  Analyze  the  WDA  modeling  results. 


4.  Analyze  the  CTA  modeling  results. 


Element  of  Cognition 

Design  Seed  Theme 

Support  Hypotheses 

Mental  models 

Specific  display  entity  (present  the 
stimulus  that  is  the  ‘organizing 
principle’  for  a  mental  model  and  thus 
immediately  trigger  that  entire  mental 
model,  rendering  it  available  for 
problem  solving  and  decision  making 
or  action).  This  display  entity  must  be 
based  on  a  good  understanding  of 
the  organization  of  information  in 
long-term  memory 

Operators  will  be  able  to  infer 
more  about  a  situation  from  a 
single  display  object 

Attentional  resources 

Minimization  of  workload,  better 
distribution  of  tasks  across  different 
channels,  reduction  in  need  for 
complex  mental  calculations,  possibly 
through  reducing  the  demand  on 
working  memory,  by  creating  external 
representations  of  information  that 
would  otherwise  need  to  be  stored  in 
working  memory  to  be  used  in  the 
calculation 

Operator  activities  will  be  more 
accurate  and  complete 

Decision  making 

Presentation  of  information  in  a  form 
that  can  be  readily  used;  system¬ 
generated  ‘starting  points’  to  kick  start 
the  operator’s  decision  making  or 
allow  them  to  focus  on  complicating 
factors.  This  includes  the  explicit 
display  of  emergent  properties  in  a 
representational  aid 

Operators  will  exhibit  more 
efficient  and  accurate  decision 
making,  as  measured  from  the 
conscious  receipt  of  new 
information  to  execution  of  some 
activity 

Working  memory 

Data  fusion;  external  ization  of 
information  that  would  otherwise 
need  to  be  held  in  working  memory 

Operators  will  have  more  spare 
working  memory  capacity 

Situation  awareness 

Presentation  of  overview  information 
with  the  opportunity  to  drill  down  for 
more  detailed  information;  suitable  for 
command  role,  all  the  way  down  to 
individual  operators 

Operator  will  exhibit  better 
situation  awareness 

Communication 

Large  shared  display,  integration  of 
systems,  re-location  of  some  systems 

Operators  will  engage  in  more 
accurate  communication 

Collaboration 

Large  shared  display,  graphical  forms 
of  communication 

Operators  will  work  more 
efficiently  together  to  develop 
solutions 

Table  4.  Mapping  elements  of  cognition  to  design  seed  themes  and  associated  support  hypotheses 


The  contribution  of  each  of  these  stages  to  the  process  is  discussed  in  more  detail  below.  The 
first  two  stages  listed  above  allowed  identifying  general  design  themes  from  SME  statements, 
and  provided  the  basis  of  their  cognitive  justifications.  By  comparison,  the  last  two  stages 
produced  more  specific  design  infonnation  (e.g.,  specific  support  requirements,  nature  of  support 
to  be  provided,  operator  interactions).  This  process  led  to  an  extensive  set  of  specific  design 
seeds  being  developed. 

We  emphasize  that  the  focus  of  the  discussion  in  this  section  is  on  the  process  we  followed  to 
develop  the  design  seeds  and  on  how  consideration  of  the  model  levels  or  elements  in  the  WDA 
and  CTA  modeling  led  to  proposals  for  specific  types  of  design  seeds.  Also,  we  limit  the 
discussion  to  only  some  aspects  of  the  model  analysis.  A  detailed  exposition  of  the  entire  model 
analysis  and  the  design  seeds  that  emerged  from  mapping  SME  statements  onto  the  ADS  and  the 
Decision  Ladder  would  require  more  space  than  the  present  paper  permits. 

4.1  Consideration  of  potential  difficulties 

Each  SME  statement  referred  to  a  task  or  some  specific  observation  they  found  noteworthy  about 
a  task.  Analysts  considered  the  task  or  the  observation  to  detennine  where  the  most  likely 
human  demands  and  difficulties  lay.  This  was  done  from  a  broad  consideration  of  the  likely 
perceptual,  cognitive,  metacognitive  and  collaborative  work  demands  operators  must  deal  with. 
For  instance,  might  a  specific  task  be  demanding  because  of  the  difficulty  for  the  operator  to 
notice  crucial  information?;  or  that  there  are  many  things  to  keep  track  of?;  or  that  it  requires  the 
comparison  and  transformation  of  information  that  varies  in  terms  of  its  modality  (e.g.,  visually, 
aurally)?;  or,  might  it  simply  be  that  the  task  is  time  consuming  at  a  time  when  the  operator  is 
already  under  time  pressure? 

The  process  by  which  analysts  considered  potential  difficulties  associated  with  SME  statements 
could  certainly  have  been  pursued  further  by  following  a  deliberate  line  of  questioning  with  the 
SMEs  themselves.  However,  for  a  variety  of  reasons  (the  internalization  of  tacit  knowledge  and 
skills,  insufficient  time,  etc.),  we  found  it  more  effective  for  analysts  to  make  these 
considerations  themselves.  Usually,  this  was  inextricably  tied  to  mapping  potential  operator 
demands  or  difficulties  along  its  cognitive  dimensions.  We  treat  this  next. 

4.2  Mapping  the  cognitive  basis  of  difficulties 

A  number  of  themes  were  readily  apparent  when  identifying  computer-based  design  seeds  based 
on  cognitively  related  difficulties.  Although  a  considerably  more  extensive  list  than  this  was 
actually  developed,  we  provide  some  examples  in  Table  4  above  for  illustration.  The  full  list 
was  developed  from  a  consideration  of  the  Wickens  model  of  human  information  processing  [9] 
(see  Fig.  3  below).  The  table  above  links  some  element  of  cognition  with  the  design  seed  theme 
that  seemed  to  be  predominant,  and  briefly  describes  the  support  hypothesis  that  can  be 
associated  with  the  theme.  As  a  means  to  an  end,  we  found  that  considering  SME  statements 
from  the  perspective  of  potential  cognitive  difficulties  was  successful  in  quickly  generating  a 
potentially  rich  vein  of  design  proposals  in  the  form  of  design  seed  themes.  However,  to  pin 
down  the  specifics  of  design  seeds  we  turned  next  to  the  WDA  and  CTA  mapping  results. 

4.3  Consideration  of  the  WDA  modeling 

The  Abstract  Function  level  of  a  work  domain  refers  to  the  underlying  laws  and  principles 
governing  activities  in  that  work  domain.  The  closest  element  of  cognition  to  the  Abstract 


Function  is  novel  problem  solving;  the  operator  takes  what  he  knows  generally  about  an  area  and 
applies  it  to  a  new  situation  without  any  known  and  defined  processes  and  procedures.  Design 
seeds  for  this  level  in  the  Abstraction  Hierarchy  must  therefore  permit  flexible  styles  of  working. 
Indeed,  design  seeds  at  the  Abstract  Function  level  must  only  bound  the  extent  of  permissible 
activity  in  accordance  with  the  underlying  laws  and  principles,  leaving  the  operator  free  to  act  in 
anyway  he  wants  beyond  these  laws  and  principles. 

A  number  of  design  seeds  were  identified  that  allow  the  operator  to  work  flexibly  to  solve  some 
new  problem.  One  of  the  simplest  was  the  provision  of  a  free  text  entry  field  that  can  be 
associated  with  a  contact  and  written  to  a  database.  This  is  because  it  is  impossible  to  anticipate 
all  the  information,  or  the  nature  of  that  information,  an  operator  may  want  to  record  about  a 
contact.  Consequently,  the  provision  of  a  free  text  field  (in  addition  to  the  strictly  bounded  data 
fields)  in  which  to  write  anything  presents  an  underlying  principle  (“you  can  record  whatever 
interesting  fact  about  a  contact  you  want”)  but  does  not  impose  a  process  on  to  it  (“it  doesn’t 
matter  what  that  infonnation  is”).  Many  other  design  seeds  were  developed  that  present 
information  in  such  a  way  as  to  reduce  the  load  on  the  operator’s  working  memory  while 
engaging  in  novel  problem  solving.  This  could  be  done  via  specific  display  entities  or  via 
composite  display  metaphors  that  provide  a  general  visualization  of  the  conceptual  constraints  on 
operator  activity. 

The  Generalized  Function  level  refers  to  the  processes  adopted  in  the  work  domain.  The  closest 
elements  of  cognition  at  this  level  are  the  mental  models,  scripts  and  schemas  that  feed  decision 
making,  situation  awareness  and  other  higher-level  cognitive  functions.  A  large  number  of 
design  seeds  were  identified  at  this  level  of  abstraction.  Many  were  automated  support  functions 
such  as  the  drawing  of  the  most  direct  navigable  route  between  two  points.  This  supports  the 
simple  process  of  planning  a  route  by  proposing  a  basic  route  to  start  with.  The  operator  could 
then  more  easily  engage  in  the  ‘what  iffing’  that  should  accompany  planning  because  they  can 
focus  on  the  other  considerations  rather  than  the  route  itself.  Operators  also  consider  specific 
tactics  for  specific  situations.  These  comparison  processes  could  easily  be  supported  with  an 
unobtrusive  support  tool  that  leaves  the  operator  free  to  generate  a  more  elaborate  solution.  One 
example  would  be  a  representational  aid  that  links  information  to  enable  comparisons  of  the 
mission,  the  constraints  and  the  entities  in  the  work  environment. 

The  last  level  from  our  selection  discussed  here  is  the  Physical  Function  level  of  a  work  domain, 
which  refers  to  the  entities  that  exist  in  the  work  domain  (e.g.,  contacts  in  local  and  operational 
environments)  and  their  capabilities.  The  closest  element  of  cognition  to  this  level  is  Situation 
Awareness  (SA)  so  that  by  designing  to  address  this  level,  we  are  attempting  to  improve  SA. 
The  design  seeds  identified  focus  on  the  visualization  and  display  of  each  entity  in  the  work 
environment  and  its  capabilities.  For  instance,  some  design  seeds  refer  to  monitors  for 
divergences  from  expected  behaviour.  Other  seeds  refer  to  databases  of  common,  expected  or 
repeated  behaviours  that  can  assist  the  contact  identification  process.  The  objective  behind  the 
latter  seeds  is  to  use  knowledge  of  the  contacts’  existence,  observations  of  their  behaviours,  and 
‘best  guesses’  of  their  capabilities  to  determine  who  they  are  and  provide  some  sort  of  contact 
recognition  support.  By  doing  this  job  for  the  operator,  the  system  is  freeing  up  time  and  effort 
to  be  directed  toward  developing  better  courses  of  action  and  getting  inside  an  opponent’s 
decision-action  cycle. 


In  summary,  it  was  possible  to  draw  parallels  between  the  Abstraction  Hierarchy  and  elements  of 
cognition,  and  develop  specific  design  seeds  at  each  level  of  the  hierarchy  from  the  mapping  of 
SME  statements  onto  their  level  of  abstraction  in  the  Abstraction  Hierarchy.  We  also  attempted 
this  for  the  Part-Whole  Decomposition  of  the  WDA  modeling,  but  we  have  found  that  its 
principal  contribution  to  the  development  of  design  seeds  has  so  far  been  largely  indirect.  For 
example,  its  consideration  suggested  the  necessity  to  tailor  the  level  of  presentation  of  and  access 
to  the  information  in  a  seed  to  the  decomposition  level  at  which  it  is  to  be  used.  Information 
serving  higher  levels  is  more  likely  to  present  an  overview,  allowing  the  operator  to  drill  down 
into  topics  of  interest.  Information  serving  lower  levels  is  more  likely  to  present  the  detail 
initially,  permitting  the  formation  of  the  overview  from  the  detail. 

4.4  Consideration  of  the  CTA  modeling 

The  CTA  modeling  also  resulted  in  a  number  of  design  seeds  being  identified.  However,  the 
nature  of  the  analysis  meant  that  it  was  more  difficult  to  correlate  design  seeds  with  specific 
data-processing  activities  or  states  of  knowledge  in  the  ladder.  To  overcome  this,  we  aggregated 
steps  in  the  Decision  Ladder  in  a  similar  manner  (see  Fig.  2  above)  to  that  in  which  the  Wickens 
model  of  human  information  processing  [9]  (see  Fig.  3  below)  aggregates  information 
processing.  The  ladder  was  split  into  three  stages,  corresponding  to  the  stages  of  perceptual 
encoding,  central  processing,  and  responding.  We  also  added  a  ‘working  memory’  stage  to  form 
a  bridge  between  perceptual  encoding  and  central  processing. 


When  conducting  the  analysis  of  the  results  of  mapping  SME  statements  onto  the  ladder,  it 
became  apparent  whether  the  emphasis  of  the  associated  operator  activity  lay  at  an  early,  middle 
or  late  stage  of  information  processing.  Predictably,  those  statements  judged  to  rest  largely  at 
the  early  stage  of  information  processing  related  to  perceptual  encoding  resulted  in  specific 
design  seeds  that  focus  on  bringing  a  stimulus  to  the  operator’s  attention,  or  retaining  that 
stimulus  in  a  location  for  the  operator  to  access  and  use  quickly  and  easily  in  decision-making 
and  problem-solving  processes.  Practically,  these  design  seeds  are  auditory  or  visual  cues, 
followed  by  displaying  objects  that  convey  the  new  information  so  that  the  operator  remembers 
that  it  exists  without  running  the  risk  of  it  being  forced  out  of  working  memory. 


The  next  stage  in  the  mapping  described  above  is  ‘working  memory’.  This  stage  suggested 
design  seeds  in  the  fonn  of  some  external  representational  aid  (i.e.,  in  a  display)  that  offloads  the 
burden  on  the  operator  of  remembering  the  information.  Traditionally,  this  would  be  a  list  or  set 
of  discrete  display  objects.  However,  this  work  has  suggested  composite,  synergistic  display 
objects  that  are  a  result  of  data  fusion,  the  consideration  of  the  infonnation  being  conveyed,  the 
cognitive  operation  to  be  perfonned  on  that  information,  and  the  mental  model  to  be  triggered. 
These  features  can  help  overcome  working  memory  limitations  and  move  information  processing 
seamlessly  from  perceptual  encoding,  through  working  memory,  to  central  processing. 

We  discuss  here  one  more  stage,  the  central  processing  stage  of  infonnation  processing.  This 
stage  encompasses  the  realization  of  the  current  system  state,  the  knowledge  of  what  state  the 
system  needs  to  be  in,  and  the  resolution  of  any  ambiguity.  In  this  work  environment  there  can 
be  a  great  deal  of  ambiguity.  The  resolution  of  this  ambiguity  and  the  determination  of  what  the 
system  state  should  be  represent  significant  challenges  to  automated  systems.  The  Decision 
Ladder,  in  contrast  to  the  Abstraction  Hierarchy,  provides  more  detail  about  the  precise  aspect 
that  should  be  addressed  by  the  design  seed.  For  example,  in  central  processing  how  can  the 
operator  determine  the  system  state  or  resolve  ambiguity?  What  infonnation  is  required  by  the 
operator  to  arrive  at  the  next  state  of  knowledge?  What  process  is  adopted  by  the  operator  to 
arrive  at  a  new  state  of  knowledge? 

This  latter  point  is  perhaps  the  most  illuminating  with  respect  to  identifying  design  seeds  from 
CTA  modeling.  This  analysis  identifies  leaps  and  shunts  (‘shortcuts’)  in  the  Decision  Ladder. 
Interestingly,  our  analysis  also  identified  additional  shortcuts  that  do  not  conform  to  the  strict 
definitions  of  leaps  and  shunts  [3],  Design  seeds  were  identified  to  accommodate  expert 
operator  ‘shortcut’  processes.  As  a  simple  example  of  this  in  the  scenario  SMEs  developed  in 
the  CDM  interviews,  when  tasked  with  escorting  a  tanker  operators  already  knew  that  they  were 
not  with  the  tanker  and  therefore  had  to  plot  a  course  to  the  tanker.  In  turn,  this  involved 
assembling  a  large  array  of  information  for  consideration.  One  design  seed  to  address  this  would 
be  to  automatically  link  mission  objectives  with  ongoing  work  activities  and  have  the  system 
monitor  other  systems’  states  and  invoke  the  appropriate  subroutines  or  decision  support  tools. 

Many  of  the  design  seeds  that  resulted  from  analyzing  the  CTA  models  focused  on  the 
automation  of  work  activities;  for  instance,  automation  of  route  plotting,  automation  of 
determining  what  weapon  system  to  use  in  response  to  threats,  automation  of  searching  for 
things  that  might  affect  a  plan.  In  comparison  to  the  WDA,  the  CTA  resulted  in  design  seeds 
that  were  much  more  focused  on  taking  the  active  problem-solving  and  decision-making  role 
away  from  the  operator.  This  possibility  may  not  always  be  the  most  appropriate  use  of  the 
analysis,  as  this  may  deskill  operators  and  render  them  less  able  to  conduct  novel  problem 
solving.  However,  by  focusing  on  supporting  the  operator  in  achieving  the  various  activities  in 
the  Decision  Ladder,  rather  than  focusing  on  actually  automating  them,  Decision  Ladders  were 
also  found  to  support  a  design  approach  that  is  complementary  to  the  human  operator. 

5.  Moving  from  Design  Seeds  to  Interface  Concept 

Traditionally,  the  identification  of  design  seeds  that  help  jumpstart  a  design  process  is  an  implicit 
process,  subjugated  to  the  development  of  a  support  concept.  However,  by  explicitly  identifying 
the  ‘small’  kernels  (i.e.,  the  seeds)  that  comprise  a  support  concept,  a  system  developer  can 
consider  them  individually  for  the  manner  in  which  they  complement  the  overall  concept.  This 


eliminates  the  addition  of  functionality  to  a  support  concept  during  the  design  process  that  can 
often  result  in  subtly  disjointed  elements  of  a  supposedly  ‘integrated’  design. 

Considering  all  the  design  seeds  produced  in  this  work  has  led  to  the  development  of  a  coherent, 
integrated  interface  concept.  The  interface  concept  represents  the  manifestation  of  several 
design  seeds  that  were  identified  during  the  analysis  discussed  above.  Figure  4  illustrates  only 
one  component  from  that  interface  concept.  This  specific  component  was  developed  to  support 
MTPC  work  on  a  FIALIFAX  Class  frigate.  Some  details  on  this  component  have  previously 
appeared  in  [5].  We  provide  a  summary  here. 


Figure  4.  Time -based  data  display  component  of  interface  concept 


The  time-based  data  display  in  Fig.  4  was  designed  to  allow  operators  to  keep  track  of  data  (e.g., 
inputs  from  the  Electronic  Warfare  Supervisor  (EWS),  radar,  communications  intercepts,  visual 
means)  across  time  and  spot  trends  and  possible  links  through  the  graphical  linking  (via  lines)  of 
related  information.  Operators  can  ctrl+click  different  information  ‘points’  to  associate  different 
data  in  the  time-based  window,  and  this  will  automatically  be  entered  to  a  database  and 
presented  in  the  target  information  display  (a  different  display  component)  when  any  point 
associated  with  that  target  is  selected.  Lines  can  be  suppressed  through  use  of  a  ‘Show  Links’ 
toggle  to  permit  the  user  to  reduce  clutter  on  their  display.  Links  are  also  shown  on  the  situation 
display  (another  display  component).  The  time  of  appearance  of  data  (a  point  on  this  display) 
will  also  be  automatically  entered  into  the  database.  Operators  can  click  and  drag  out  a  vertical 
‘time’  cursor  from  the  ‘NOW’  position  to  any  position  along  the  time  axis  and  replay  the 
appearance  of  data  from  any  point  at  various  faster  than  real-time  speeds.  Replay  also  replays  in 
the  situation  display.  Clicking  on  any  piece  of  data  (points)  in  the  time-based  data  display  will 
show  links  (if  links  are  suppressed),  highlight  related  (target)  entry(ies)  on  the  ORB  AT  window 
(another  display  component),  and  the  situation  display  and  will  invoke  the  corresponding 
information  display. 

An  in-depth  exposition  of  how  this  display  concept  and  others  emerged  from  the  process 
described  in  this  paper  will  be  covered  in  future  publications.  In  reality,  the  process  of  moving 
from  design  seeds  to  a  coherent  and  integrated  support  concept  can  be  summarized  as  an 
evolutionary  ‘conversation’  between  Human  Factors  specialists,  software  and  hardware 
engineers,  and,  crucially,  operators.  The  exact  manifestation  of  a  design  seed  will  depend  upon 
many  things,  such  as  Human  Factors’  best  practices  along  a  number  of  dimensions,  software  and 
hardware  capabilities,  operator  preferences,  and  new  empirical  work  focusing  on  the  fit  between 
different  aspects  of  a  display.  For  example,  composite  display  objects  in  which  data  is  fused  will 


rest  on  research  into  the  modes  of  information  display  required,  and  the  manner  in  which  these 
requirements  can  be  integrated. 

6.  Conclusions 

The  experience  of  identifying  design  seeds  using  the  process  described  in  this  paper  was 
interesting  from  a  number  of  perspectives.  First,  design  seeds  and  a  viable,  integrated  support 
concept  were  developed  within  three  weeks  of  data  collection  with  SMEs.  That  is  to  say,  the 
data  was  collated  and  structured,  Work  Domain  Analysis  and  Control  Task  Analyses  were 
conducted,  design  seeds  were  identified  and  the  manner  in  which  they  could  be  put  together  to 
create  an  integrated  support  concept  was  considered.  These  activities  were  performed  by  four 
analysts.  This  result  is  in  marked  contrast  to  the  impression  of  an  onerous  process  that  is  often 
communicated  by  the  CWA  literature. 

The  next  point  refers  to  the  large  number  of  design  seeds  developed  during  this  work.  In  fact, 
regardless  of  the  ‘style’  of  statement  taken  from  SMEs  (i.e.,  task  or  observation),  it  was  possible 
to  develop  a  design  seed  for  every  single  one.  In  the  development  of  new  systems,  this  would 
provide  ample  inspiration  for  design  teams  to  create  innovative  and  useful  new  systems.  The 
ready  development  of  design  seeds  lends  credence  to  the  notion  that  CWA,  and  specifically 
WDA  and  CTA,  are  merely  means  to  an  end;  that  the  important  aspect  is  the  development  of 
useful  design  outcomes.  As  such,  the  reliability  and  utility  of  the  data  and  the  analysis  may  in 
the  end  be  of  less  importance  than  what  they  tell  us  about  design  seeds  and  what  we  can  do  with 
them. 

This  leads  to  a  final  point  specifically  about  design  seeds,  that  their  identification  should  mark  a 
return  to  data  collection.  During  this  work,  an  ‘open-ended’  approach  to  data  collection  was 
taken  in  order  that  SMEs  could  define  their  own  problems  and  resultant  design  seeds  would  be 
based  on  SMEs’  own  implicit  knowledge  of  what  comprised  the  difficult  elements  of  their  work. 
However,  this  means  that  the  data  collection  sessions  necessarily  covered  a  great  deal  of  ground, 
with  limited  opportunities  to  gather  all  the  detail  to  fully  develop  all  seeds.  As  a  result,  the  data 
was  ‘patchy’  at  times  in  terms  of  a  consistent  level  of  detail  and  uncovering  all  aspects  of  an 
activity.  Because  of  inconsistent  data,  analysis  could  also  potentially  be  inconsistent.  Design 
seeds  could  still  be  identified  though,  and  many  of  them  can  be  considered  quite  insightful. 
However,  in  order  to  fully  ‘flesh  out’  a  design  seed,  it  is  indeed  necessary  to  return  to  the  SMEs 
and  investigate  the  design  seed’s  area  of  application  in  meticulous  detail.  This  was  not  pursued 
during  this  work,  but  would  be  a  logical  next  step. 

Although  the  focus  of  this  work  has  been  to  develop  design  seeds  of  a  computer-based  nature, 
the  process  developed  was  also  found  to  be  worthy  of  exploration  for  its  potential  extension  to 
the  development  of  seeds  addressing  an  even  broader  set  of  design  considerations,  including 
training  and  operator  organization. 

The  resource  and  time  constraints  inherent  in  this  work  have  meant  that  the  utility  of  CWA- 
based  approaches  to  system  development  has  been  stringently  tested,  and  the  result  for  the 
process  we  developed  and  followed  based  on  our  exploratory  design  framework  has  been 
positive.  This  work  has  also  challenged  the  perspective  that  is  normally  adopted  in  a  CWA: 
CWA  was  developed  primarily  with  a  view  to  the  revolutionary  design  of  a  work  domain  (i.e., 
not  constrained  by  an  existing  instantiation  of  the  domain);  however  this  work  has  applied  it  to 
the  evolutionary  design  of  an  individual’s  task  (i.e.  not  the  whole  work  domain)  in  an  existing 


work  domain.  As  far  as  we  know,  this  may  also  be  the  first  work  to  use  CDM  to  do  a  CWA;  the 
first  to  use  the  ADS  and  Decision  Ladders  to  model  SME  ‘statements’;  and  the  first  to  use 
Decision  Ladders  to  model  shipboard  Operations  Room  activities.  Finally,  this  work  also 
appears  to  be  the  first  to  convincingly  demonstrate  a  concrete,  traceable  progression  from  data, 
to  work  analysis  and  modeling,  to  the  identification  of  design  seeds  and  support  hypotheses, 
based  on  a  consideration  of  CWA  results  and  of  cognition. 
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v/How  do  we  Design  to  Support  C2  Work? 


ognitive  activity  distributed  across  multiple,  interacting  actors 


Evolving  interconnected  flow  of  activities,  varying  phases  and  tempos 


•  New  missions,  new  operational  contexts  are  leading  to  evolving 
cognitive  and  collaborative  demands  and  increasing  complexities 

•  Growing  pressures  for  agile  and  adaptive  responses 


Human  expertise  and  capacity  for  adaptation  play  an  increasingly 
vital  role  in  this  environment 

Few  design  frameworks  aimed  at  developing  tools  to  support 
operator  adaptation 

Investigating  a  work-centred  design 
framework  incorporating  a  form  of  work 
analysis  known  as  Cognitive  Work  Analysis 


Routine 


Unfamiliar 
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Application  areas 

•  Maritime  Tactical  Picture  Compilation 

•  Tactical  Planning  and  Response  Management 


Application  focus 

•  Support  existing  work  processes  with  computer-based 
solutions  (evolutionary  changes) 

•  High-level  consideration  of  a  ‘blue  sky’  design  solution 
(revolutionary  changes) 
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Support  Hypothesis 


ognitive  Work  Analysis  (CWA)  Framewor 


Increasing 

Constraint 


Phases  of  CWA  Kinds  of  Information  Modeling  Tools 

Work  Domain  Analysis 

Purpose  and 
structure  of  work 

domain 

Abstraction- 
decomposition  space 
(ADS) 

Control  Task  /  Activity 
Analysis 

Goals  to  be  satisfied, 
decisions/cognitive 
processinq  req'd 

Decision  ladder  (DL) 
template 

Strategies  Analysis 

Ways  that  processing 
can  be  executed 

Information  Flow  Map 

Social  Organisation  and 
Cooperation  Analysis 

Who  carries  out  work 
and  how  it  is  shared 

Annotations  on  all  the 
above 

Competencies  Analysis 

Kinds  of  mental 
processing  supported 

Skills,  Rules  and 

Knowledge  model 

CWA  concentrates  on  modeling  intrinsic  behaviour-shaping 
constraints  on  work 

Formative  focus  promotes  concepts  to  support,  flexible, 
adaptive  operator  behaviour: 

higher-level  control  (situation  independent)  vs  lower-level 
control  (situation  dependent) 

Defence  R&D  Canada  -  Atlantic  •  R  &  D  pour  la  defense  Canada  -  Atlantique 


Knowledge  Elicitation 

Earlier  work  has  used  various  approaches:  Subject  Matter 
Experts  (SMEs)  walk-through  with  pre-scriptea  scenarios; 
SMEs  work  through  pre-scnpted  scenarios  m  Navy  trainer 

Work  reported  here  based  on  semi- structured,  but  open- 
ended,  interviews  with  teams  of  SMEs 

CIT  vs.  CDM 

-  Originally  looked  at  Flanagan’s  Critical  Incident  Technique 
(CIT) 


•  CIT  not  specifically  designed  for  retrospective  interviews 

•  CIT  looks  at  a  large  corpus  of  critical  incidents  -  hundreds  (or 
thousands) 

-  Critical  Decision  Method  (CDM)  designed  for  retrospective 
interviews,  focuses  on  fewer  decision  points  and  cognition  bases 
of  judgement  and  decision  making 

Chose  Critical  Decision  Method  (CDM;  Klein,  Calderwood 
and  MacGregor,  1989). 

3  intact  operator  teams  involved  over  2  days 
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CDM  Interview  Steps 


•  Step  1  -  Identify  an  incident 

•  Step  2  -  Unstructured  incident  approach 

•  Step  3  -  Sequence  of  events  construction 


Step  4  -  Planning  (Decision)  point  identification 

Step  5  -  Decision  point  probing:  triggers/cues, 

information,  goals,  options,  situation  awareness, 
etc. 
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Data  Reduction 

Collate  notes  from  data  collection  sessions 
Integrate  notes  from  different  analysts 
Supplement  with  audio  data  where  necessary 
Enter  into  Excel  spreadsheet 

Finalize  chronological  description  of  the  scenarios  described  by  the 
SMEs 


expanded  sequence  of  events  based  on  responses  to  decision  point 
probes 

structured  according  to  high-level  tasks,  lower  level  activities  or 
observations 

derived  directly  from  SME  statements 


Escort  vessel  through  Strait  of  Hormuz ”  (Planning  incident) 
•  •  • 

“Consider  risks  in  escort”  (Event/high-level  task) 

•  •  • 

“Determine  the  trade-off  between  defending 
High-Value  Unit  and  putting  Own-Ship  at  undue  risk” 
(Lower-level  activity/ observation) 
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~\7  Work  Modeling 


General  Modeling  Procedure  Followed 


4  analysts  independently  mapped  each  SME  statement  onto  the 
CWA  model  templates: 


Need  for  mechanisms  to  ensure  I  napping  reliability:  exemplars, 
guidelines 

Analysts  met  at  end  to  reconcile  Snapping  differences  and  agree  on  a 
final  mapping 
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-yWork  Modeling:  Work  Domain  Analysis  (WD 

A  WDA  models  the  work  domain  in  the  form  of  an 
Abstraction-Decomposition  Space 

ADS  built,  including  a  set  of  exemplars  for  each  ADS  cell 
Analysts’  mapping  of  SME  statements,  using  exemplars  as  a 
check  of  internal  consistency 
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Functional  Purpose 

What  was  the  work  domain  desiqned  to  do? 

Abstract  Function 

What  are  its  underlying  laws  or  principles? 

Generalized  Function 

What  are  the  processes  that  are  involved? 

Physical  F  unction 

What  entities  are  involved  and  what  are  their  capabilities? 

Physical  Form 

What  is  the  physical  appearance  and  location  of  an  entity? 

Geopolitical, 

weather, 

geophysical, 

etc 
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V  WDA  (cont’d) 


An  example: 


“Determine  the  trade-off  between  defending  High-Value  Unit  and 
putting  Own-Ship  at  undue  risk” 
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^  Work  Modeling:  Control  Task  Analysis  (CTA) 


•  A  CTA  maps  out  control  tasks  in  terms  of 
data-processing  activities  and  states  of 
knowledge  using  a  decision  ladder  (DL) 
template 


Coding  scheme  developed  and  guidance 
provided  to  analysts  for  mapping  onto  the 

Analysts’  mapping  of  SME  statements 


DL 


Defence  R&D  Canada  -  Atlantic  •  R  &  D  pour  la  defense  Canada  -  Atlantique 


#14 


_  -CTA  (cont’d) 

R 07  , 

An  example: 


“Consider  risks  in  escort”  (Event/high-level  task  of  ship’s  tactical  coordinator) 


-  “Determine  the  trade-off  between  defending  High-Value  Unit  and  putting 
Own-Ship  at  undue  risk” 
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>7  Developing  Design  Seeds 


A  design  seed  is  some  specific  and  relatively  independent 
support  concept  for  some  specific  aspect  of  the  work; 
could  be  realized  in  a  number  of  forms;  jumpstarts 
exploratory  design  process 

Will  be  accompanied  by  a  specific  hypothesis  about  the 
nature  of  its  support 

Focusing  on  design  seeds  as  an  intermediate  step  allows 
designer  to  consider  them  both  individually  and 
collectively  to  see  how  they  complement  a  proposed 
overall  integrated  support  concept 

4  stage  process  followed,  based  on  analysis  of  each  SME 
statement 

•  Identify  potential  operator  difficulties 

Map  cognitive  basis  of  difficulties 

Analyze  WDA  modeling  results  (Abstraction- 
Decomposition  Space) 

Analyze  CTA  modeling  results 


Central 

Perceptual  Encoding  Proceeding 

I  .  »  4  »  4 


Responding 


SENSORY 

INPUT 


N-jf 

'A3 


Working  : 
Memory  | 

T'T 

Long-Term  Memory 


RESPONSE 

OUTPUT 


> 


General  design  themes  and  support 
hypotheses 


Kxccuti-  ] 


IH 


Specific  design  information: 
specific  support  reqmts,  nature  + 

of  support,  etc.) 
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>7  Developing  Design  Seeds:  A  WDA  Example 


Abstraction 

Hierarchy 

Cognition 

High-level  Design 
Seeds 

Functional 

Purpose 

Significant  paradigm 
shift 

Monitor  to  determine 
when  functional 
purpose  must  change 

Abstract 

Function 

Novel  problem 
solving 

Display/monitoring  of 
work  constraints  but 
permitting  flexible 
ways  of  working 

Generalized 

Function 

Mental  models, 
schemas,  scripts,  etc, 
feeding  decision 
making  and  situation 
awareness 

Automated  support 
functions 

Physical 

Function 

Situation  awareness 

Display  of  entities, 
their  capabilities  and 
their  behaviour 

Physical  Form 

Perception 

Appearance,  location, 
layout 

Wo*, 

Operational 

Environment 

Local 

System 

Sub 

System 

Component 

Component 

ii 

Determine 

the  tradeoff 
between 
defending 
High-Value 

Unit  and 
putting  Own- 
Ship  at  undue 
nsk" 

function 

Phyrical 

form 

•  Support  for  novel  problem 
solving  and  decision  making 

•  Make  constraints  apparent  (e.g., 
display  of  dynamic  relationships 
between  risk  considerations) 
(Abstraction  Hierarchy) 

•  Since  operator  must  consider 
local  environment  (i.e.,  not  just 
Own- Ship),  extend 
considerations  to  HVU 
relationships  (e.g.,  databases, 
adaptive  selection  and  re¬ 
calibration  of  relationships) 

•  Led  to  the  development  of  a  risk 
management  assistant  support 
concept  for  tactical  planning 
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Summary  and  Conclusions 


Knowledge  elicitation  based  on  open  data  collection  allowed  SMEs  to  identify  and  frame  the 
problem  space  themselves 

Data  structuring  was  achieved  within  3  days  of  data  collection  by  a  team  of  4  analysts;  WDA  and 
CTA  analyses  and  production  of  design  seeds  took  analyst  team  12  days 
Analysis  and  modeling  was  hindered  by  the  variability  in  the  data: 

-  further  phase(s)  of  knowledge  elicitation 

-  development  of  a  formal  and  precise  grammar  for  representing  the  data 

Identification  of  design  seeds  and  specific  support  hypotheses  was  arguably  the  most  successful 
part  of  the  process:  a  design  seed  was  generated  for  each  SME  statement 

Resource  and  time  constraints  in  this  project  meant  that  the  utility  of  a  CWA-based  approach  was 
severely  tested,  yet  not  found  to  be  as  onerous  as  the  literature  suggests 

Although  CWA  was  developed  primarily  for  revolutionary  design,  it  was  found  to  be  also  effective 
from  an  evolutionary  design  perspective 

To  date  only  a  limited  evaluation  of  design  concepts  has  been  undertaken 

-  feedback  from  Command  Team  members  to  a  crude  mock-up  of  a  risk  management  assistant  during  a 
Canadian  Navy  Task  Combat  Readiness  Operation 

A  number  of  firsts  in  this  work  (as  far  as  we  know) 

-  use  of  CDM  to  do  a  CWA 

-  use  of  CWA’s  Abstraction  Decomposition  Space  and  Decision  Ladders  to  model  SME  statements  for  a 
shipboard  Operations  Room 

'  -  has  led  to  a  convincing  demonstration  of  a  traceable  design  thread  from  actual  SME  data,  to  work  analysis 
and  work  modeling,  to  identification  of  design  seeds  and  support  hypotheses 
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